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Foreian lanquaqe tests fall into two classes, according to their purpose.. I he 
first class, tests used for the control of instruction, may be achievement or diagnostic 
tests. The second class of tests, used in the control of a persons career. may be 
concerned with what the sublet can do. or what he should be able to do in the fvlure. 
The temporal distinction is less important than the mapr functional o^exacuy^the 
same test can serve as a diagnostic test before some material i s taught, and a= an 
achievement test after. Similarly, proficiency tests are generally used a* predictor o 
future performance. The author suggests using a functional definition of levels wh c . 
would test ability to operate easily and effectively m specified sociolmguistic situations 
(rather than attempting to characterize levels 
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of the second class, it remains a serious difficulty, for no way has yet been found to 
characterize knowledge of a language with sufficient ^ P^ c,5lon to guaran ee e 
validity of the items included or the type of tests used.(AMM) 
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Language Testing— The Problem of Validation 

Bernard Spolsky 



Foreign-language tests fall naturally 
into two classes according to the pur- 
poses for which they are used. In 
the first class are tests used for the 
control of instruction. They are the 
concern of the classroom teacher who 
wishes to find out how effective his 
teaching and the students’ learning 
have been or to discover what needs 
to be taught. The second class are 
tests used in the control of a person’s 
career. Used by administrators or 
counsellors, they are intended to help 
make decisions about someone’s quali- 
fications for a given task or about the 
type of training he should follow. 

Each of these classes may be fur- 
ther divided according to the tem- 
poral relation of the test to its goal. 
Tests of the first class concerned with 
testing what has been taught are 
achievement tests; those concerned 
with what is about to be taught are 
diagnostic tests. Tests of the second 
class concerned with what the subject 
can do now are achievement tests; 
those concerned with what he should 
be able to do in the future are predic- 
tive tests. But this temporal distinc- 
tion is less important than the major 
functional one; exactly the same test 
can serve as a diagnostic test before 



* This paper was presented at the TESOL 
Convention, March 1968. 
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some material is taught and as an 
achievement test after it. Similarly, 
proficiency tests are generally used as 
predictors of future performance. 

This functional classification agrees 
with one that can be made on opera- 
tional grounds: tests of the first class 
are relatively simple to prepare and 
straightforward to interpret, while 
tests of the second class involve serious 
theoretical and practical difficulties in 
preparation, interpretation, and es- 
pecially, in validation. Why this is 
so becomes clear if we consider the 
steps to be followed in preparing a 
test. 

Take the preparation of a test of 
the first type, an achievement or di- 
agnostic test, to be used by a class- 
room teacher either before she starts 
a unit or chapter in the textbook or 
after she has finished. The starting 
point is the syllabus, with its list of 
items to be learned in the unit. The 
purpose of the test will be to decide 
how many of the items on the list have 
been mastered by the students. For 
our example, let us assume that we 
have an elementary class in English 
as a second language; we wish to test 
their knowledge of vocabulary, and 
our syllabus is defined by Lesson Elev- 
en of Book One of English for Today . 1 
Notice that the first, and in many 
ways, most important task of test 
writing has been done: the syllabus 
(or textbook in this case) gives us a 
list of the sixteen new words in the 
lesson. There is no point in our going 



1 English for Today , ed. William R. Slager 
(New York: McGraw-Hill, 1962). 
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beyond the list unless of course we 
wish to test words previously taught. 
From it, we select the words to be 
tested. If we have time, we can put 
every word in the test, but there is 
no need, for using some appropriate 
techniques, we can choose a represen- 
tative sample. Next, we have to de- 
cide on the testing technique we are 
going to use. It is here that we are 
called on to define more precisely what 
it means to “know vocabulary”; we 
need to translate the general term into 
a more precise one. Here are some 
possible operational definitions, each 
describing a possible technique: 

(a) When presented with a word 
on the list, the student taking the test 
should say, “I know it” or “I don't 
know it.” 

(b) When presented with a word 
on the list, he must select which one 
of a group of definitions is appropri- 
ate: 

glass — something you drink 
out of 

something you paint 
with 

something you draw 
with 

(c) When presented with a picture 
of the object named by a word on the 
list he indicates its name: 

glass 

cup 

bottle 

(d) 'When presented with a picture, 
the student must write down what it is. 

There are of course many other tech- 
niques, 2 but these may be considered 
a representative sample. Of course, 

* See, for details, Robert Lado, Language 
Testing (New York: McGraw-Hill, 1961), 
or Rebecca Valette, Modem Language 
Testing: A Handbook (New York: Har- 
court, Brace and World, 1967). 



they each raise minor problems. The 
first, (a), might not be considered a 
normal sort of test, but it is likely to 
be a most useful technique with teach- 
ers and students who are cooperating 
closely in the learning process. The 
second and third raise a special prob- 
lem: when multiple choice items are 
given, the student should know the 
meaning of the incorrect answers as 
well as the correct one; otherwise, the 
choice is unduly limited. For example, 
in (c) above, bottle would be a bad 
distractor, for the word is not intro- 
duced in Lesson Eleven (or in fact in 
Book One). Similarly, (b) has a bad 
definition; the word drink comes in 
Lesson Sixteen; in Lesson Eleven, all 
you do with glasses is wash them. A 
more serious problem in choosing a 
test technique is deciding whether it 
is a valid representation of the skill 
we want to test. Is there a serious dif- 
fer .,ce between being able to recog- 
nize a definition and being able to give 
a definition? The former technique 
is easy to mark, the latter takes much 
longer. But it is quite easy to try out 
all the different techniques, and decide 
for ourselves whether they correlate 
so well that we only need to use one 
in the future.* Once we have decided 



• The Interpretive Information for the 
Test of English as a Foreign Language 
(Educational Testing Service, 1967, revised 
January 1968) for example describes an 
interesting comparison of the scores on ‘he 
“writing” section (a set of multiple-choice 
questions) with the scores of the same stu- 
dents on a set of essays graded by a team 
of examiners. The correlation is .74, which 
is close enough to suggest that the saving 
in tim* is worthwhile, unless of course we 
are planning to interpret the scores as 
though they had 100 percent validity. And 
on this see Paul Holtxman’s paper in 
NAFSA Studies and Papers, English Lan- 
gauge Series, Number 18: ATESL Selec- 
ted Conference Papers (1967). 
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on the items and technique, the rest of 
the task of preparation is simple. And 
interpretation is straightforward, too. 

As long as the test is a representative 
sample of items, its result will say, 
“This student scored sixty percent on 
the test; he knows sixty percent of 
the words on the list.” If the test is 
a diagnostic test, we will know what 
words need be taught; if an achieve- 
ment test, we will know how effective 
our teaching has been. What has made 
test preparation and interpretation so 
simple has been that we have been 
able to ask a question to which there 
is a quantifiable answer. We have not 
asked whether or not the student 
knows English vocabulary, but rather 
how many of the words on this list he 
knows. Our results are clear, for they 
say he knows a given percentage of 
the words in Lesson Eleven of the text- 
book. 

Basic to this relative simplicity was 
the existence of a list of items to be 
tested. Clearly, such lists are not 
available for all tests used in control 
of instruction. But it is equally clear 
that effective teaching depends on the 
availability of clear specifications. Nor- 
mally, we have a syllabus or textbook 
or both, with lists of vocabulary, 
grammatical structures, etc. With 
such a syllabus or textbook, test 
making is straightforward. A control 
of instruction test is concerned with 
the question, “Have the items listed 
in the syllabus or textbook been 
learned to some criterion level?” It 
is not concerned with what should be 
learned. It would be wrong to include 

in a test of this class items that are 
not included in the syllabus. 

When we say then that an achieve- 
ment test is not a good one, we are 



referring to its inability to test a de- 
fined body of material; we are not 
s a ying anything about what should 
constitute that material. That is the 
task of the syllabus or textbook writer. 
Now, there are clearly cases when the 
distinction between test writer and 
textbook writer are blurred. One such 
case is when the test writer is trying 
to evaluate achievement in something 
that has not in fact been specified. 

He then has to do the textbook writer’s 
job of specification before he can pre- 
pare an achievement test. This hap- 
pens when one has a set of materials 
that can be listed as items and pat- 
terns, but one wishes to test the ability 
of the students to know more than the 
items or patterns they have been 
taught. For example, one may wish 
to find out about a student’s ability 
to speak naturally on a topic other 
than those he has been trained for in 
memorized dialogues, or to use pat- 
terns with words other than those 
included in the pattern drills. But in 
such cases, we are really moving out 
of the realm of achievement tests, and 
into the area of proficiency, the second 
class of tests. These cases in fact set 
the limit; the first class in its purest 
form consists of tests defined not only 
functionally but also operationally— 
functionally, in that they are used in 
the control of instruction, and opera- 
tionally, in that they are tests prepared 
on the basis of specifications of be- 
havior or items that have been pre- 
pared, independently of the test, as 
part of the development of materials, 
textbooks, and syllabus. 

The second class of foreign-language 
tests is defined functionally as tests 
used primarily in the control of a sub- 
ject’s career. They serve to. majca 
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dgments possible on such questions 
« 

1. How well will the subject do 
; learning foreign languages in gener- 
, or one foreign language in particu- 
r? Should he be advised (permitted, 
icouraged) to study a language? 
bould his employer (or the govern- 
ment, or the armed forces) invest time 
nd money in his studying a lan- 
juage? 

[ 2. How well does the subject per- 
wrm in the given foreign language? 
f he needs to use the language in 
ovemment or other service, will he 
ie successful? If he is a graduate stu- 
dent in a given field, will he be able 
o read books in the foreign language? 

Tests aimed to handle the first set of 
questions are predictive tests; their 
ask is to make some sort of judgment 
possible on the question of the stu- 
lent’s language-learning aptitude, and 
will need to make available informa- 
tion on any factors that will be rele- 
vant to language learning. This type 
bf test sets many basic problems about 
the nature of second language acquisi- 
tion, but will be left out of considera- 
tion in this paper. 4 Here, we shall be 
concerned with tests intended to an- 
swer questions of the second sort, pro- 
ficiency tests . . 

Fundamental to the preparation of 
valid tests of language proficiency is 
a theoretical question of what it means 
to know a language. There are two 
ways in which this question can be 
answered. One is to follow what 
John Carroll 5 has referred to as the 



integrative approach, and to accept 
that there is such a factor as overall 
proficiency. The second is to follow 
what Carroll called the discrete-point 
approach: this involves an attempt to 
break up knowing a language into a 
number of separate skills, and further 
into a number of distinct items making 
up each skill. We are using the overall 
approach when we give a subjective 
evaluation of the proficiency of a 
foreign speaker of our language. In 
such cases, we usually do not refer 
to specific items that he has or hasn’t 
mastered but to his ability to function 
in a defined situation. We do not say, 
“He is unable to distinguish between 
the phonemes /i/ and iy/,” but rather 
something like “He doesn’t know 
enough English to write an essay, but 
he seems to be able to follow lectures 
and to read his textbooks without 
much trouble.” The key assumption 
of the discrete-point approach is that 
it is possible to translate sentences of 
the second type into a list of sentences 
in the first, and the key requirement 
for discrete-point testing is that we 
could quantify “He knows the words 
on this list.” 

Detailed instructions on how to 
prepare tests like this are given in the 
books by Lado and Valette referred to 
earlier. Drawing in particular on the 
powers of techniques developed by 
taxonomic linguistics to describe in 
detail the surface structure of lan- 
guages, Lado shows how it is possible 
to construct tests that permit very 
fine discrimination of the strengths 
and weaknesses of foreign-language 



♦ For a disc ussion of this problem, see 
Paul Pimsleur, ‘Testing Foreign Language 
Learning,” Trends in Language Teaching, 
ed. Albert Valdman (New York: McGraw- 
Hill, 1966). 



•John B. Carroll, “Fundamental Consid- 
erations in Testing for English Language 
Proficiency of Foreign Students,” Testing 
(Center for Applied Linguistics, 1961). 
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learners. Basic to Lado’s approach is 
a theory calling for systematic descrip- 
tion of the surface structure of the 
language being learned, combined with 
comparison with the language of the 
learner; it leads to a notion that tests 
as well as teaching material should be 
based on contrastive analysis, and pre- 
pared accordingly. Using these tech- 
niques, it is possible to develop tests 
that may be scored objectively (al- 
though some studies have raised some 
questions about the type of ques- 
tion used ) 8 and the results of which 
lead to such precise interpretation 
tion as “the subject confuses medial 
/!/ and /r/.” Tests of this nature are 
obviously of very great value in the 
control of instruction, whether as di- 
agnostic or achievement tests. 

But we must ask whether such an 
approach, assuming that all we have 
to do is to list all the items, permits 
us to characterize overall proficiency. 
If so, overall proficiency could be con- 
sidered the sum of the specific items 
that have been listed and of the spe- 
cific skills in which they are testable. 
To know a language is then to have 
developed a criterion level of mastery 
of the s kills and habits listed. There 
are rather serious theoretical objections 
to this position. First, a discrete-point 
approach assumes that knowledge of 
a lan guag e is finite in the sense that 
it will be possible to make an exhaus- 
tive list of all the items of the lan- 
guage. Without this, we cannot show 
that any sample we have chosen is 
representative and thus valid. We 
must then argue for selection on the 

•See for instance, Eugtae Brifcre, “Test- 
ing the Control of Parts of Speech in FL 
Compositions,” Language Learning, XIV, 
1 & 2 (1964). 



basis of functional necessity. This in- 
volves defining the functional load of 
the ability to distinguish between a 
pair of phonemes or of the ability to 
recognize the appropriateness of a 
given verb form. To do this, we would 
have to collect a list of minimal pair 
utterances in which the distinction is 
vital, but there turn out to be very 
few real minimal-pair situations, that 
is, situations where a single linguistic 
difference in a given situation will 
lead to complete misunderstanding. 

I have been told for instance the true 
story of a foreign lady speaking to her 
Italian maid: she asked to have the 
meat ( came ) brought to the table, but 
had it given to the dog (cane) instead; 
rather strong punishment for speaking 
an r-less dialect. The rarity of such 
situations is a result (and theoretical 
cause) of the redundancy of natural 
languages." Thanks to redundancy, we 
can communicate satisfactorily with- 
out knowing any given item in a lan- 
guage. This is most obvious in the 
area of vocabulary, where it is quite 
clear how many of the words in the 
dictionary are unknown to the average 
native speaker; it is true also in the 
area of phonology, otherwise speakers 
of different dialects of the same lan- 
guage would never be able to under- 
stand each other. It is probably not 
true in the case of many syntactic 
rules, but many of these are likely 
to turn out to be universal, and so ir- 
relevant to foreign-language testing. 
More important, though, is the fact 
that syntactic rules are untestable un- 
less fleshed out with vocabulary and 
phonology or spelling. 



» For a brief account of this, see John B. 
Carroll, Language and Thought (Englewood 
C liffs : Prentice-Hall, Inc., 1964). 
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All of this suggests the impossibility 
of characterizing levels of knowing a 
language in linguistic terms, that is, 
as mastery of a criterion percentage of 
items in a grammar and lexicon. A 
more promising approach might be to 
work for a functional definition of 
levels: we should aim not to test how 
much of a language someone knows, 
but test his ability to operate in a 
specified sociolinguistic situation with 
specified ease or effect. The prepara- 
tion of proficiency tests like this would 
not start from a list of language items, 
but from a statement of language 
function; after all, it would not be 
expected to lead to statements hke 
“He knows sixty percent of English,” 
but “He knows enough English to shop 
in a supermarket.” 

Functional statements of language 
proficiency may take various forms. 
One of the most thorough examples of 
a fairly complete scale is that prepared 
by the Foreign Service Institute for 
the classification of officers of the U.S. 
State Department. These Absolute 
Language Proficiency Ratings, as they 
are called, involve a division into lan- 
guage skills (reading, writing, speak- 
ing, and comprehending) and a nu- 
merical rating for each. The numerical 
ratings are .generally described by a 
brief title, and range from “elemen- 
tary,” through “working” and “pro- 
fessional” to “native or bilingual.” 
For each level of each skill there is a 
short description, again emphasizing 
skill. For example, to receive the 
rating S-3, one must be “able to speak 
the language with sufficient structural 
accuracy and vocabulary to satisfy 
representation requirements and han- 
dle professional discussions within a 
special field.” There is then a longer 



description, suggesting the type of 
language-learning experience that is 
associated with the level.* 

Starting with functional statements 
of this sort (and there should be little 
problem in preparing such descrip- 
tion for each of the situations in 
whia. proficiency tests are used), the 
language tester’s problem is to find a 
reliable, valid, and economical method 
of rating a subject’s proficiency in 
these terms. The first question is one 
of strategy. The discrete-point ap- 
proach implies that it is possible to 
give a linguistic description of each 
level, to list the words and grammar 
needed to achieve this, but this is not 
possible either in theory or practice. 
The practical approach followed in the 
past has been to decide in some ad hoc 
way (the opinion of teachers, for in- 
stance) on the sort of items to be 
tes ted and the sort of test to use, but 
even though such a test can be made 
extremely reliable, it proves impossible 
to show its validity with sufficient pre- 
cision to justify interpretations or im- 
provements.* A more helpful strategy 
is to prepare proficiency tests in two 
stages. For the first stage one must 
forget considerations of expense and 
time. Expensive tests, using panels 
of trained judges, and having the sub- 
ject function in situations of the sort 



•The Absolute Language Proficiency 
Ratings are described in a number of mimeo- 
graphed circulars. More accessible is the 
sample quoted by John Carroll in his 
article in Foreign Language Annal* I. 2. 
(December 1967), and the description by 
Frank Rice in the Linguietic Reporter (May 
1959). 

•This problem has been discussed, among 
other places at a seminar held at the 19 o' 
Conference of the National Association for 
Foreign Student Affairs, the proceed^Sso* 
which have been published in ATESL Se- 
lected Conference Paper*. 
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described in the rating scales, should 
first be developed as yardsticks. The 
second stage then involves taking 
cheaper procedures, of whatever kind, 
and correlating them with the more ex- 
pensive measures. The degree of corre- 
lation will show the value of the ad hoc 
tests and make clear the degree of 
doubt that must be kept in their in- 
terpretation. 

The exact nature of these more prac- 
tical tests is not important; one would 
presume that they would be s imil ar 
to many of the tests presently being 
used,” but they would permit of greater 
confidence in use, greater possibility 
of improvement (for we could then be 
in a position to speak about improving 
the validity of an objective test), and 
greater refinement in interpretation. 
It is probable that we would be able 



to develop simpler tests (e.g., the over- 
all proficiency test using redundancy 
I have been working on 11 ), and so ulti- 
mately justify the expense of the 
validation procedures. 

The central problem of foreign-lan- 
guage testing, as of all testing, is valid- 
ity. With tests of the first class, used 
by classroom teachers in the control 
of instruction, this problem is not seri- 
ous, for the textbook or syllabus writer 
has already specified what should be 
tested. With tests of the second class, 
it remains a serious difficulty, for we 
have not yet found a way to character- 
ize knowledge of a language with suf- 
ficient precision to guarantee the valid- 
ity of the items we include or the 
types of tests we use. 



“John Carroll, for instance, has investi- 
gated the correlation between the FSI Ab- 
solute Languag# Proficiency Ratings and 
the MLA Foreign Language Proficiency 
Tests lor Teachers and Advanced Students. 



“Bernard Spolsky, Bengt Sigurd, Mass- 
hito Sato, Edward Walker, and Catherine 
Arterbum, “Preliminary Studies in the De- 
velopment of Techniques for Testing Over- 
all Second Language Proficiency," Inter- 
national Review of Applied Linguistice, 
(in press).. 






